Skip to content

Events listing, deletion and archival optimizations#11443

Open
winterhazel wants to merge 7 commits into
apache:mainfrom
scclouds:listevents-optimizations
Open

Events listing, deletion and archival optimizations#11443
winterhazel wants to merge 7 commits into
apache:mainfrom
scclouds:listevents-optimizations

Conversation

@winterhazel

Copy link
Copy Markdown
Member

Description

The listEvents and deleteEvents APIs execute some non-optimized queries, which results in very long response times in environments that have many events.

Some optimizations were performed in these two workflows in order to reduce response times. The listing now uses a covering index, and does not perform a unecessary DISTINCT anymore. The deletion is now performed in batches. The batch size is defined by the global setting delete.query.batch.size.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI
  • test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

How Has This Been Tested?

Events listing

In a test environment with 2 million event entries, I called listEvents multiple times passing different combinations of account, archived, domainid, duration, enddate, entrytime, id, keyword, level, projectid, resourceid, resourcetype, startdate, startid and type. For almost all the listings, I got a more reasonable response time (between 0.1 and 3 seconds depending on the filters) compared to before (30+ seconds). The only exception is the keyword parameter, which is not optimizable with a index because it performs a column LIKE "%keyword%" for 3 different columns.

Access control

I called listEvents and verified that:

  • root admins can list all events;
  • domain admins can only list events from their domain;
  • users can only list their events.

Events deletion

  1. I assigned a value to delete.query.batch.size, enabling batch delete;
  2. I called deleteEvents to remove all events before 2025-01-01. Then, I verified that they were removed successfully;
  3. I assigned a value to event.purge.delay, enabling the automatic deletion of events older than 15 days. After the deletion task executed, I verified that events older than 15 days were removed successfully.
Access control

I called deleteEvents and verified that:

  • root admins can delete any events;
  • domain admins can only delete events from their domain;
  • users can only delete their events.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants